Induction of Multiscale Temporal Structure
Learning structure in temporally-extended sequences is a difficult
computational problem because only a fraction of the relevant
information is available at any instant. Although variants of back
propagation can in principle be used to find structure in sequences,
in practice they are not sufficiently powerful to discover arbitrary
contingencies, especially those spanning long temporal intervals
or involving high order statistics. For example, in designing a
connectionist network for music composition, we have encountered
the problem that the net is able to learn musical structure that
occurs locally in time--e.g., relations among notes within a
musical phrase--but not structure that occurs over longer time
periods--e.g., relations among phrases. To address this problem,
we require a means of constructing a reduced description of
the sequence that makes global aspects more explicit or more readily
detectable. I propose to achieve this using hidden units that
operate with different time constants. Simulation experiments
indicate that slower time-scale hidden units are able to pick up
global structure, structure that simply can not be learned by
standard back propagation.
Retrieve Paper (pdf)